Augmented reality holds great promise for providing information relevant to a person's current environment. With augmented reality, visual images are added to a scene of an environment currently being viewed by a person. The visual images are synthetic graphical images (also referred to as “synthetic graphics” hereinafter) that are generated by a computer and added to a real-world scene. For example, a synthetic graphic may be overlaid or superimposed in front of the real-world scene. A synthetic graphic may also be inserted into the real-world scene such that the inserted visual image of the synthetic graphic appears to be partially behind a real-world object at some depth in the field of vision. With a finely-orchestrated insertion of a synthetic graphic, the synthetic graphic can appear to be integrated with the real-world scene, which can provide increased realism or enable annotations that do not block important visual aspects but still clearly indicate what visible object is being annotated.
A synthetic graphic may be helpful, humorous, entertaining, and so forth. For example, an augmented reality system can display directional arrows and textual instructions over streets or down hallways to help a person navigate to a requested destination. In another example, recommended tolerances or detailed installation instructions for an automotive part are displayed to help a car mechanic In yet another example, an augmented reality system can project the results of a planned structural remodel or the introduction of new furniture into a real-world view of an interior of a building. A synthetic graphic can further be used for amusement, such as displaying amusing comments or recent social media communications, for example, over the heads of friends and coworkers. In another example, an electronic “buddy” can be introduced into the surrounding space of a child to act as a playmate, tutor, or guide. Thus, augmented reality has the potential to positively impact many areas of life by enabling the addition of synthetic graphics into real-world scenes.
An augmented reality system may be configured in a variety of ways. For example, the augmented reality system may be configured as glasses to be disposed in front of a user's eyes, goggles that cover the eyes, another type of head-mounted display, a display screen, a video projector, or some combination thereof. These systems also include functionality to generate the synthetic graphic for visual presentation in conjunction with the real-world scene.
Augmented reality systems are typically built as one of two main types: see-through systems supporting a direct-view or camera-mediated systems supporting an indirect view. With systems of the former type, a person looks through a display screen presenting a synthetic graphic and also may directly view the real-world scene disposed behind the display screen. With systems of the latter type, a person looks at a display screen to see both the synthetic graphic and the real-world scene as jointly presented on the display screen. With both types of augmented reality systems, some approach is instituted to composite a synthetic graphic with the real-world scene to create a composited view. Unfortunately, a number of technological hurdles prevent the attainment of the full potential of augmented reality with either the direct-view approach or the camera-mediated approach.
With a camera-mediated augmented reality system, a person's view is conventionally constrained to a single-layered display screen presenting the world as captured through the use of a camera, but the display screen is physically opaque to the real-world from the perspective of the eye of the person viewing the display screen. In other words, the person's view is indirect because the person does not receive light originating from the real-world. Instead, a camera with an image sensor is exposed to the real-world scene, and one or two miniature display screens display the real-world scene by piping in data from an image sensor of the camera. A camera-mediated augmented reality system is analogous to looking through the viewfinder of a modern camcorder in which a liquid crystal display (LCD) screen presents a view of what the image sensor of the camcorder is currently “seeing” while lighting or focus indications are superimposed over the scene being recorded.
Accordingly, camera-mediated augmented reality systems allow for accurate compositing because the combining of the real-world scene and the synthetic graphics is performed electronically and then displayed together on a single-layered display screen. Unfortunately, camera-mediated systems degrade a person's view of the real-world environment due to resolution and latency constraints of the camera, image processing components, and limitations of the display screen, itself. A camera-mediated augmented reality system also becomes dark and completely opaque if the system experiences a power failure.
In contrast, with a direct-view augmented reality system, a person's eyes are exposed to light rays originating from real-world objects. A direct-view system includes a display screen having one or more display layers configured for insertion between the person's eyes and a real-world scene as directly viewed. An example of a display layer for a display screen that is at least partially see-through is an emissive display layer. An emissive display layer can be primarily transparent to the light originating from real-world objects, but the emissive display layer can also selectively present images that are sufficiently opaque so as to be discernable to the human eye. Thus, direct-view augmented reality systems use advanced optics incorporated into one or more display layers to create the illusion of a transparent display at a fixed distance in front of the viewer as super-imposed over a real-world scene. Because real-world objects are viewed directly, the real world is seen with minimal latency.
Unfortunately, direct-view augmented reality systems also suffer from a number of drawbacks, such as a narrow field of view. Furthermore, the synthetic graphics added to a real-world scene are subject to the resolution constraints of a display sub-system, which can include multiple display layers of a display screen along with associated processing capabilities. Additionally, once synthetic graphics are super-imposed by a direct-view augmented reality system, a person's ability to properly focus on the synthetic graphics can be compromised by anatomical characteristics of the human eye. For example, presenting a synthetic graphic in a manner to cause the eye and the mind to locate the synthetic graphic at a desired depth in the real-world view is difficult. This is especially so when the eye and the mind of a person are trying to combine light originating approximately an inch from the eye with light originating from real-world objects that are tens, hundreds, or more feet away from the person.
To facilitate accurate compositing of close-up synthetic graphics and far-away real-world objects, a freeform optics prismatic lens can be used as a display layer of a display screen in a direct-view augmented reality system. A freeform optics prismatic lens presents synthetic graphics with proper focus cues to help a person see the displayed synthetic graphics without blurriness and at the intended depth in the field of view. Integrated light-weight eye tracking can be employed to estimate where or on what object a person's eyes are currently focusing to determine a focal depth.
A challenging problem, however, remains with the introduction of occlusive synthetic graphics to direct-view augmented reality systems. A synthetic graphic added to a real-world scene occludes a person's view of one or more real-world objects. Although the occluding synthetic graphic blocks the view of some real-world objects, the occluding synthetic graphic is nevertheless desirable because the graphic provides information to the person, such as a description or other annotation of an object in the real-world scene. On the other hand, distracting image artifacts are undesirable. Direct-view augmented reality systems continue to be problematic because such systems produce distracting image artifacts in conjunction with the presentation of occluding synthetic graphics. An example of an image artifact is a dark halo around a displayed synthetic graphic. Such image artifacts result from multiple see-through display layers, the light originating from the real-world scene, or the interaction of multiple layers and incoming light.
One relatively simple approach, at least from an optical perspective, for introducing occlusive synthetic graphics into a real-world scene utilizes a conventional stack of LCD layers. The stack of LCD layers can both attenuate the real world and provide synthetic graphics in the foreground along with proper focus cues. Unfortunately, such devices have low spatial resolution and require computationally expensive optimization processing to display the synthetic graphics. Moreover, for the stacked LCD layers to gain angular resolution, the resolution of the LCD layers is increased. But LCD layers of sufficiently high resolution can blur the transmitted view of the real world due to diffraction effects. Employing three such LCD layers that are stacked on top of each other further exacerbates this problem. Additionally, if a conventional stack of LCD layers is used to attenuate the view of a person's surroundings to support viewing a synthetic graphic, a dark halo is visible around the synthetic graphic. Alternative optical systems have been proposed, but the alternative optical systems are both bulky and complex. The alternatives also compromise other desirable aspects of an augmented reality system, such as good contrast.